slimTrain---A Stochastic Approximation Method for Training Separable Deep Neural Networks

Deep neural networks (DNNs) have shown their success as high-dimensional function approximators in many applications; however, training DNNs can be challenging general. DNN is commonly phrased a stochastic optimization problem whose challenges include nonconvexity, nonsmoothness, insufficient regularization, and complicated data distributions. Hence, the performance of on given task depends crucially tuning hyperparameters, especially learning rates regularization parameters. In absence theoretical guidelines or prior experience similar tasks, this requires solving series repeated problems which time-consuming demanding computational resources. This limit applicability to with nonstandard, complex, scarce datasets, e.g., those arising scientific applications. To remedy training, we propose \tt slimTrain, method for reduced sensitivity choice hyperparameters fast initial convergence. The central idea slimTrain exploit separability inherent architectures; that is, separate into nonlinear feature extractor followed by linear model. allows us leverage recent advances made large-scale, linear, ill-posed inverse problems. Crucially, weights, does not require rate automatically adapts parameter. our numerical experiments using approximation tasks surrogate modeling dimensionality reduction, outperforms existing methods recommended hyperparameter settings reduces remaining hyperparameters. Since operates mini-batches, its overhead per iteration modest savings realized reducing number iterations (due quicker convergence) need solved identify effective